Practical recommendations from a multi-perspective needs and challenges assessment of citizen science games

Citizen science games are an increasingly popular form of citizen science, in which volunteer participants engage in scientific research while playing a game. Their success depends on a diverse set of stakeholders working together–scientists, volunteers, and game developers. Yet the potential needs of these stakeholder groups and their possible tensions are poorly understood. To identify these needs and possible tensions, we conducted a qualitative data analysis of two years of ethnographic research and 57 interviews with stakeholders from 10 citizen science games, following a combination of grounded theory and reflexive thematic analysis. We identify individual stakeholder needs as well as important barriers to citizen science game success. These include the ambiguous allocation of developer roles, limited resources and funding dependencies, the need for a citizen science game community, and science–game tensions. We derive recommendations for addressing these barriers.

R1: Ethical considerations of citizen science games were discussed, but this article on that specific topic was not cited: https:  of citizen science games) • We added citations to Schrier on the ethics of CSGs as suggested.

R1: Why did you use purposeful sampling? Was this because of the small sample size of CGSs or ones you have access to?
• Our aim was twofold. For the HCI study, we aimed to gather a representative sample from among many of the most popular and well-known CSGs. This was, as you say, limited by who we had access to speak to, but we sought to sample from multiple games without sampling too heavily from any specific one. For the ethnographic study, we took a deeper approach to ensure that our findings are grounded in lived truths. However, no change was made to the text because we believe this is adequately captured in the nature of purposeful sampling.

R1
: Was there just one person conducting the thematic analysis or was this a shared task among the researchers? It's stated but still not clear why the secondary analysis was used.

Themes vs. topics --can you give examples of what you mean by this?
• We added a clarification that the initial thematic analysis was performed only by the first author. • As described in the paper: the initial thematic analysis of the HCI study was leading to topics rather than themes, which is why the joint analysis was performed (to produce themes). And, as stated in the first line of Section 2.3, the joint analysis was performed by the first two authors. Regarding topics vs. themes, we added a note to refer to Braun and Clarke 2019 which helps disambiguate this: topics are data domains (e.g., "Scientists' Perspectives") while themes are "patterns of shared meaning underpinned by a central organising concept" (Braun and Clarke 2019, p. 1). Our four themes in Section 4 are examples of this. • We added a clarification on the discussion of topics vs. themes to refer to the works of Braun and Clarke.

R1: co-laborative --> collaborative? Or is this a term that is undefined/jargon that should be defined further?
• We added quotes around "co-laborative" with a brief definition to accompany the cited reference.
R1: It's not clear why the thematic analysis was used in the first place, and it is not clear enough why the second series of methodologies were used.
• We added further explanation to the purpose of the first thematic analysis and to the purpose of the second qualitative analysis.
R1: This sentence, and the surrounding sentences, are a bit confusing. "Following an ethnographic and grounded theory approach data collection and analysis did not present independent successive but alternating phases." It's also confusing that you have some games used in the first approach, and then a subset in the second, and why it was a subset. I find the methods descriptions and rationales confusing and glossed over too much.
• We added more detail to clarify the grounded theory approach and on the methodological approach of the ethnographic study.

R1: I am glad you explained the games. Should that be moved up earlier in the methods section? Could there be a chart of games and which methods you used with which?
• We added a table to show which games were included in which studies. (We kept the section where it is because we felt it was important to first introduce how they are being studied.)

R1:
Recommendations are great to see. I would include some with valuing the professions of game design, development, and community management more readily, with a plan of how to fund these as a professional, rather than through student work or interns.

R1: Can they get funding for a vendor?
• We added elaboration to recommendations of funding.

R1: Also, I know this is an area of expertise for some of you, but it is problematic when it says
"gamified" throughout the paper--these are games (CSGs) not gamified content. To me, they should be considered full-fledged games from the outset.
• Here we use this term to mean making a game based on real science, rather than creating gamified content. We agree that CSGs are, for the most part, a gameful approach rather than a gamified one, but there is no better term for saying "turning a scientific task into a game" than "gamifying."

R1: I like the table for recommendations. Could you separate it out as recommendations for development team, scientific community, educational community, etc?
• For purposes of scope we have chosen to limit our recommendations to the development team only. As noted at the top of Section 5, we refer to prior literature for recommendations to other communities and stakeholders.
R2: A more thorough description of the projects, with an explanation of the type of gaming employed (leader boards, collaborative, first-person, exploration etc.), and perhaps an investigation on how these types correlate to the opinions of stakeholders -this could be a whole follow-up paper however! • We added more detail to the games studied, ensuring that our descriptions of the games studied included the type of gaming employed and the discipline of focus. Unfortunately, as R2 says, correlations with the opinions of stakeholders would be an entire follow-up paper.
R2: Perhaps a section on the disciplines involved in each project, how they relate to the school subjects, and if that correlates with any of the opinions of the students and/or teachers involved.
• We added detail to ensure that each project's discipline is clear. Unfortunately, there is insufficient data to make claims about correlations between game disciplines and how they relate to school subjects. Notably, we also added a section on data availability to ensure that our manuscript complies with the PLOS data policy.
We thank the reviewers for their time and hope that these revisions are satisfactory.